Subjective and Objective Quality Evaluation of Lightly Distorted Synthetic Images

Measuring visual quality, as perceived by human observers, is becoming increasingly important in a large number of applications in which humans are the ultimate consumers of visual information. To aid in the developing of objective image quality assessment (IQA) algorithms, many natural image databases have been developed that contain the subjective ratings of the images by human observers. Subjective quality evaluation data is less available for synthetic images such as those commonly encountered in viewing video games or animated movies. In addition, a variety of powerful full-reference, reduced-reference and no-reference IQA algorithms have been proposed over the past decade or so for natural images, but their performance has not been evaluated on synthetic images. In this paper we (1) conduct a series of subjective tests on a new publicly available Embedded Signal Processing System (ESPL) Synthetic Image Database containing 500 distorted images (20 distorted images for each of the 25 original images) in 1920 × 1080 format, and (2) evaluate the performance of more than 50 publicly available IQA algorithms which we use on the new database. We also conducted a human study on the database, collecting 26000 individual human ratings and associated mean opinion scores for each image to statistically evaluate full-reference, reducedreference and no-reference IQA algorithm performance. We find that natural scene statistics can be successfully used to predict the perceptual quality of synthetic scenes and we also identify statistical features that can be used to represent distortions of synthetic images.